We study formation of ostracism in a society from a game theoretical perspective. The dynamics of group formation is complicated in that the choices of the individuals and the form of the groups mutually affect each other in the process. A suggested simple model shows that individual efforts to increase his/her own sense of belonging is responsible for both growth of groups and creation of an outcast. Once a person happens to get behind in synchronizing with others, tendency to alienate him may grow among others, possibly making him left out in the end. Alienating minority occurs even when there is a penalty for disliking and people are encouraged to favor others. Considering that the target is accidentally picked, we can understand ostracism as an inherent part of the group formation, rather than a result of specific discrepancy among people. Another finding is that a single individual who seeks for unconditional unification of the society (“philanthropist”) likely invites his/her own isolation from the society, while the existence of such person generally promotes coalition of others.